1. Field of the Invention
This invention relates generally to testing in Java based environments, and more particularly to an enhanced method for same virtual machine mode testing capable of self-recovery.
2. Description of the Related Art
As the use of software in performing daily tasks rapidly increases, assessing software reliability through software testing has become an imperative stage in the software development cycle. As is well known, software testing is used to find and eliminate defects (i.e., bugs) in software, which if undetected, can cause the software to operate improperly. One method for performing testing in a Java environment is via a test harness. FIG. 1 is a block diagram showing a prior art testing configuration 100, wherein the test harness and each test are executed in separate virtual machines.
By way of background, Java is an interpreted language that utilizes a Java Virtual Machine (JVM) as an interpreter. The JVM is a software layer that converts Java intermediate language (bytecode) into machine language. As shown in FIG. 1, a test harness is executed on a computer system 102 within a first JVM 104. In addition, a test is executed within a second JVM 106, which is separate from the first JVM 104 executing the test harness. Generally, both the first JVM 104 and the second JVM 106 are executed on a single computer system 102, although it should be noted that the prior art testing configuration can run multiple tests, each in its own JVM.
The prior art testing configuration 100 is often referred to as “other virtual machine (VM) mode” because each test is executed in a JVM 106 separate from the JVM 104 of the test harness. In operation, the test harness typically runs a test suite, which is a collection of related tests, by starting a new JVM 106 for each test of the test suite. When each test is complete, the test harness stops the JVM executing the particular test. Because each test is executed within its own JVM 106, any failure in the JVM 106 will only affect the particular test executing within that JVM 106. That is, the test harness 104 will not be affected. Other tests executing in separate JVMs also will not be affected by the failure of any one test JVM. As a result, “other VM mode” is considered a safe and reliable method for performing testing in a Java based environment.
An important task of a test harness is to provide a flexible environment for stress testing. Many techniques exist to increase stress level when testing a product. For example, special test cases can be created that are specifically designed to stress the tested system. However, in the case of Java, the stress level can be doubled by implementing “same VM mode” testing, as illustrated next with reference to FIG. 2.
FIG. 2 is a block diagram showing a prior art testing configuration 200, wherein the test harness and each test are executed in the same virtual machine. As shown in FIG. 2, the test harness is executed on a computer system 102 within a JVM 104. However, instead of starting a new JVM for each test, the test harness calls test subroutines 202 to execute the various tests of a test suite. Similar to FIG. 1, the prior art testing configuration 200 can run multiple tests within the test harness JVM 104.
The prior art testing configuration 200 is often referred to as “same VM mode” because each test is executed in the same JVM as the JVM 104 of the test harness. Because the test harness 104 is not required to start and stop separate JVMs for each test, same VM mode testing generally is six to ten times faster than “other VM mode” testing. In addition, same VM mode stresses the system in a different manner than “other VM mode” testing, generally resulting in increased stress that improves testing. For example, the HEAP configuration and the threading environment are different, that is, a different memory use pattern emerges and a different set of threads is executed. Whereas in “other VM mode” tests are isolated from the other test environments, in same VM mode environment interactions result in different thread synchronization issues and different system interrupts. As a result, increased stressing can be achieved, which results in a better test.
Unfortunately, same VM mode test configurations 200 are generally less reliable. For example, because the test harness and all the tests are executed within a single JVM 104, the remainder of the test suite will not be performed if the JVM 104 fails. For example, if a JVM executing a test suite including 10,000 tests fails on the 100th test, the remaining 9,900 tests will not be performed. Since many tests are large, and thus left executing overnight, errors may not be found until the following morning, thus increasing test time and decreasing testing efficiency.
In view of the foregoing, there is a need for systems and methods that provide the benefits of “same VM mode” testing, while having the robustness of “other VM mode” testing. The methods should provide the speed and stress increasing properties of same VM mode, and the self-recovery ability of “other VM mode.”